Methods and systems for executing tests using grouped/filtered test classes during testing of an application

ABSTRACT

Computer implemented methods and systems are provided for executing tests in a system that includes a user system and a cloud-based computing platform. The user system includes a processing system, memory and an input system that receives input parameters specified by a user of the user system. The memory can store a test class filter module executable by the processing system. Upon being executed by the processing system, the test class filter module can, based on one or more of the input parameters, group and filter test class identifiers to generate a unique test class identifier array of filtered test class identifiers that correspond to a particular subset of tests that are to be executed during testing of an application. A test execution engine can then execute the particular subset of tests corresponding to the filtered test class identifiers specified in the unique test class identifier array.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/218,334, filed Sep. 14, 2015, which is hereby included by referencein its entirety.

TECHNICAL FIELD

Embodiments of the subject matter described herein relate generally tocomputer implemented methods and systems for testing an application in acloud-based computing environment, and, more particularly, to computerimplemented methods and systems for grouping and filtering test classesand executing tests using the grouped/filtered test classes.

BACKGROUND

Many enterprises now use cloud-based computing platforms that allowservices and data to be accessed over the Internet (or via othernetworks). Infrastructure providers of these cloud-based computingplatforms offer network-based processing systems that often supportmultiple enterprises (or tenants) using common computer hardware anddata storage. This “cloud” computing model allows applications to beprovided over a platform “as a service” supplied by the infrastructureprovider.

Multi-tenant cloud-based architectures have been developed to improvecollaboration, integration, and community-based cooperation betweencustomer tenants without compromising data security. Generally speaking,multi-tenancy refers to a system where a single hardware and softwareplatform simultaneously supports multiple customers or tenants from acommon data storage element (also referred to as a “multi-tenant datastore”). The multi-tenant design provides a number of advantages overconventional server virtualization systems. First, the multi-tenantplatform operator can often make improvements to the platform based uponcollective information from the entire tenant community. Additionally,because all users in the multi-tenant environment execute applicationswithin a common processing space, it is relatively easy to grant or denyaccess to specific sets of data for any user within the multi-tenantplatform, thereby improving collaboration and integration betweenapplications and the data managed by the various applications. Themulti-tenant architecture therefore allows convenient and cost effectivesharing of similar application feature software between multiple sets ofusers.

Cloud-based applications for a cloud computing environment are typicallydeveloped by many different “users” including developers, customers,independent software vendors (ISVs), quality engineers, business units,virtual groups, sub-tenants, scrum teams, etc. An integrated developmentenvironment typically includes a collection of tools that can be used tocreate, debug, and test applications. A cloud-based application must betested by a variety of different users who belong to differentcross-functional teams, business units, sub-tenants, etc. To facilitatethis testing a wide variety of test tools and test execution engineshave been developed.

Existing test tools simply allow the user to manually select certaintests (e.g., by using a checkbox for a particular test name in a testexecution page) they want to execute, or to select all possible tests tobe executed. These approaches can be tedious and time-consuming. Itwould also be desirable to provide users with greater control over whichtests are executed during testing that pertain to the interest orownership area of a sub-tenant.

During testing of an application, “code coverage” is one particularmetric that is of interest to users who are conducting tests on theapplication. For example, a user may be interested in the amount of codecovered by certain tests of selective components and/or modules (e.g.,which lines of code are covered by a certain tests of selectivecomponents and/or modules versus which lines are not). It would also bedesirable to provide users with improved tools for computing codecoverage and presenting code coverage results that pertain to theinterest or ownership area of a sub-tenant.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

A more complete understanding of the subject matter may be derived byreferring to the detailed description and claims when considered inconjunction with the following figures, wherein like reference numbersrefer to similar elements throughout the figures.

FIG. 1 illustrates a block diagram of an example cloud-based computingenvironment in which the disclosed embodiments may be implemented.

FIG. 2 is a block diagram that shows modules executed by a user systemand their interaction with the test execution engine and data store viathe SOQL interface of FIG. 1 to execute tests using grouped and filteredtest classes in accordance with an embodiment.

FIG. 3 is a flowchart that illustrates an exemplary method for groupingand filtering test class identifiers and executing tests using thegrouped and filtered test class identifiers in accordance with anembodiment.

FIG. 4 is a block diagram that shows other modules executed by the usersystem and their interaction with the SOQL interface, tooling API, codecoverage computation unit and data store of FIG. 1 to generate codecoverage metrics using grouped and filtered source class identifiers inaccordance with an embodiment.

FIGS. 5A and 5B collectively illustrate a flowchart that illustrates anexemplary method for grouping and filtering source class identifiers andgenerating code coverage metrics using grouped and filtered source classidentifiers in accordance with an embodiment.

FIG. 6 is a screenshot that illustrates an example of a consolidatedreport that can be displayed at a user interface of the user system inaccordance with one exemplary implementation of an embodiment.

FIG. 7 is a schematic block diagram of a user system in accordance withan embodiment.

FIG. 8 illustrates a block diagram of an example environment in whichthe embodiments described herein may be implemented.

FIG. 9 illustrates a block diagram of another example environment inwhich the embodiments described herein may be implemented.

FIG. 10 illustrates a block diagram of sub-tenant process spaces thatcan be defined within one of the tenant process spaces of FIG. 9 toexplain the concept of sub-multi-tenancy in accordance with anembodiment.

DETAILED DESCRIPTION

Current testing systems do not provide the user with the ability toselectively choose a group or set of tests that they want to execute.Manual section of the certain tests is a time-consuming and tediousprocess especially when the number of tests to be executed is large.Selecting an option to run all tests is inefficient since it isunnecessarily comprehensive and time consuming when the user may notcare about the test results for many of the tests. In addition, the testresults are simply presented in a way that presents all of the testresults to the user thereby making it difficult for the user to identifywhich test results the user is interested in. Another problem is thatsome tests can take too long to execute, which can block other testsfrom running thereby delaying the user from viewing the results theywant to view. It would be desirable to improve the user's efficiency andproductivity during testing, and to give the user more flexibility inchoosing which tests to execute. It would also be desirable to presentthe results of the desired tests in a way that is easy for the user toview and interpret.

In the development of multi-tenant applications, different sub-tenants(e.g., functional groups or business units) can develop different parts(e.g., modules or components) of an application, and may therefore beinterested in code coverage metrics for different “parts” of the codebase that they are involved with. However, current tools do not providean easy way for a specific user or sub-tenant to segregate certain setsof code so that code coverage can be computed for the specific portionsof the code base that they are interested in. For example, one existingtool computes code coverage for an entire organization, and allows theuser to graphically view (line-by-line) code coverage for a given sourceclass. However, to view the code coverage metrics, the user has to view(in a UI of a developer console) each class in the organization. If theuser wants to get the code coverage metrics for a given set of classes,the user has to repeat this process for each class. This is not anoptimal use of resources and time, and especially when the entire set ofclasses is too large. The user has no easy way to choose which sourceclasses they are interested in (e.g., that are relevant to the user'steam versus the entire organization), and obtain code coverage metricsfor those source classes that are easy to view and interpret.

FIG. 1 illustrates a block diagram of an example cloud-based environment110 in which the disclosed embodiments may be implemented. FIG. 1 andthe following discussion are intended to provide a brief, generaldescription of one non-limiting example of an example environment inwhich the embodiments described herein may be implemented. Those skilledin the art will appreciate that the embodiments described herein may bepracticed with other computing environments.

The cloud-based computing environment 110 includes a user system 112that is communicatively coupled to a cloud-based computing platform 116by a network 114.

The cloud-based computing platform 116 includes a processor system 117,a data store 124 for storing data, a software application 126 that isbeing tested, a connection interface 130, an SOQL interface 140, a testexecution engine 150, a tooling API 160, and a code coverage computationunit 170 that is configured to compute code coverage. In one embodiment,the cloud-based computing platform 116 can include all of the elementsencompassed within block 116, and in other embodiments, one or more ofthe elements (e.g., code coverage computation unit 170) may beimplemented externally to the cloud-based computing platform 116.

In other embodiments, environment 110 may not have all of the componentslisted and/or may have other elements instead of, or in addition to,those listed above. For ease of illustration, FIG. 1 shows one block foreach of user system 112, processor system 117, and data store 124. Theuser system 112, processor system 117, and data store 124 may representmultiple user systems, processor systems, and data store units. Itshould be appreciated that although FIG. 1 illustrates the processorsystem 117 as a single block that this implementation is for purposes ofillustration and is non-limiting. In one embodiment, the connectioninterface 130, SOQL interface 140, test execution engine 150, andtooling API 160 can each have their own dedicated processor or processorsystem.

In one embodiment, the cloud-based computing platform 116 can be part ofan on-demand database services system that implements a cloud platform.The user system 112 is communicatively coupled to the cloud-basedcomputing platform 116 over the network 114. The cloud-based computingplatform 116 uses objects to store data, and enables a user (e.g., viauser system 112) to interact with a cloud platform to create and managedata that is stored within data store 124 as system data. For example,the user system 112 can communicate information input by a user to thecloud-based computing platform 116, and the processor system 117 canexecute the application 126 to process this information to createobjects that can then be stored in data store 124 as system data.

An object is an instance of a class. As used herein, an object can referto a structure used to store data and associated metadata along with aglobally unique identifier that allows for retrieval of the object. Asused herein, the term “class” refers to a template or blueprint fromwhich objects are created. To explain further, all objects have stateand behavior, that is, things that an object knows about itself, andthings that an object can do. A class can contain variables and methods.Variables are used to specify the state of an object, whereas methodsare used to control behavior. A class can contain other classes,exception types, and initialization code. Objects not only providestructure for storing data, but can also power the interface elementsthat allow users to interact with the data, such as tabs, the layout offields on a page, and lists of related records. Objects can also havebuilt-in support for features such as access management, validation,formulas, triggers, labels, notes and attachments, a track field historyfeature, security features, etc. Attributes of an object are describedwith metadata, making it easy to create and modify records eitherthrough a visual interface or programmatically.

In one embodiment, the cloud-based computing platform 116 may utilize adevelopment tracking system built on the Salesforce.com's force.complatform using tools such as Visualforce (an XML syntax typically usedto generate HTML) and Apex (a proprietary Java-like programming languagefor Force.com). Objects that are stored can include records associatedwith code to be integrated into the software application in order toprovide changes to the software application.

As used herein, the term “Apex” refers to a strongly typed,object-oriented programming language that allows developers to executeflow and transaction control statements on a Force.com platform serverin conjunction with calls to the Force.com API. Force.com applicationsare built using Apex and Visualforce. Using syntax that looks like Javaand acts like database stored procedures, Apex enables developers to addbusiness logic to most system events, including button clicks, relatedrecord updates, and Visualforce pages. Apex code can be initiated by Webservice requests and from triggers on objects. Apex can be used to addbusiness logic to applications, to write database triggers, and toprogram controllers in a user interface layer. Apex has a tightintegration with the database and query language, good web servicessupport, and includes features such as futures and governors forexecution in a multi-tenant environment. Additional details regardingApex will be briefly summarized below, but additional information can befound in the Apex Developer Guide, Version 36.0, Spring '16, LastUpdated: Mar. 3, 2016, which is incorporated by reference herein in itsentirety.

The Apex programming language is saved and runs in the cloud on theForce.com multitenant platform. Apex is tailored for data access anddata manipulation on the platform, and it enables users to add custombusiness logic to system events. When a developer writes and saves Apexcode to the platform, the platform application server first compiles thecode into an abstract set of instructions that can be understood by theApex runtime interpreter, and then saves those instructions as metadata.When an end-user triggers the execution of Apex (e.g., by clicking abutton or accessing a Visualforce page), the platform application serverretrieves the compiled instructions from the metadata and sends themthrough the runtime interpreter before returning the result. Theend-user observes no differences in execution time from standardplatform requests.

Testing is the key to successful long-term development and is a criticalcomponent of the development process. As such, Apex development usuallygoes hand in hand with unit test development.

Apex provides built-in support for unit test creation and execution,including test results that indicate how much code is covered, and whichparts of Apex code could be more efficient. To ensure that all customApex code works as expected, Apex code can be tested using a developerconsole to check code coverage and run Apex tests, including unit tests,functional tests, regression tests, and so on. ApexUnit is a Javaapplication that provides a powerful continuous integration tool forForce.com platform. ApexUnit includes two major components: an xUnitbased testing framework for Force.com platform, and extensive codecoverage metrics with actionable detail for Apex source code. The xUnitbased testing framework leverages the features exposed by the Force.complatform to queue and submit Apex tests (preferably a subset of targetedtests) for execution. For example, unit tests are class methods thatverify whether a particular piece of code is working properly. Unit testmethods take no arguments, commit no data to the database, send noemails, and are flagged with the testMethod keyword or the isTestannotation in the method definition. Also, test methods must be definedin test classes, that is, classes annotated with isTest. Force.comtooling API's are used for fetching code coverage results. The Force.complatform requires that at least 75% of Apex classes are covered bytesting before code can be deployed to a production system.

Referring again to FIG. 1, the connection interface 130 is an interfacethat is used to establish a communicative connection between the usersystem 112 and the cloud-based computing platform 116 over the network114. In one embodiment, the connection interface 130 can be a webservices connector (WSC) that is a code-generation tool and runtimelibrary for use with Force.com web services. The WSC uses ahigh-performing Web services client stack implemented with a streamingparser that can be used with salesforce.com APIs. In accordance with oneimplementation of the disclose embodiments, a partner connection WSDLcan be leveraged to establish connection between cloud-based computingplatform 116 and the application 126 and this connection establishescommunication channel for data/information exchange between cloud-basedcomputing platform 116 and the application 126.

The Salesforce Object Query Language (SOQL) interface 140 is aninterface between the user system 112 and other parts of the cloud-basedcomputing platform 116 including, test execution engine 150, codecoverage computation unit 170, and database(s), such as the data store124 of the cloud-based computing platform 116, which, in oneimplementation can be a Force.com data store. SOQL is speciallyoptimized version of SQL specially designed for working withSalesforce.com underlying data store. SOQL is a SQL like query languagewhich is used to interact with the data stored as tables in theForce.com database. Salesforce Web services connector provides partnerconnection utility which facilitates querying for data in Force.comdatabase using the SOQL statements. As will be described in greaterdetail below, the data store 124 can store data such as test classidentifiers, test results, source class identifiers and code coveragemetrics (among other things).

The test execution engine 150 executes various tests on a group of testclasses that are specified by a user of the user system 112. Inaccordance with the disclosed embodiments, as will be explained ingreater detail below, the test execution engine 150 receives a uniquetest class identifier array of filtered test class identifiers, andgenerates test results (e.g., test execution details) that correspond tothose filtered test class identifiers that are stored in the data store124. The test results can then be retrieved from the data store via theSOQL interface 140 and provided to the user system 112. As used herein,the term “test class” refers to a collection of test method(s) thatrepresent a test algorithm to validate the accuracy and functionality ofthe application algorithm. In a general sense, the test code pertainingto test method(s) or test classes call the source code, to validate thereturned output of source code with expected output (also known asassertion).

In accordance with the disclosed embodiments, one function of thetooling API 160 is that it serves as an interface within the cloud-basedcomputing platform 116 to a code coverage computation unit 170 that isused to compute code coverage. The tooling API 160 receives a uniquearray of filtered source class identifiers that have been specified by auser of the user system 112, and the code coverage computation unit 170uses these to compute code coverage metrics that correspond to theunique array of source class identifiers. The code coverage computationunit 170 stores the computed code coverage metrics that correspond tothe unique array of source class identifiers in the data store 124 sothat they can be retrieved by the user system 112 via the tooling API160. As used herein, the term “source class” refers to a unit of sourcecode of an application, where source code is any collection of computerinstructions, possibly with comments, written using some human-readablecomputer language. As used herein, the term “code coverage” refers to ametric used to describe the degree to which the source code of a programis tested by a particular test run/iteration. To measure what percentageof code has been tested in an iteration, one or more coverage criteriaare used. Coverage criteria is usually defined as a rule or requirement;that a test suite needs to satisfy. A program with high code coveragehas been more thoroughly tested and has a lower chance of containingsoftware bugs than a program with low code coverage. Many differentapproaches can be used to calculate code coverage; some of the mostbasic are the percent of program subroutines and the percent of programstatements called during execution of the test suite. In one testingframework code coverage numbers can be generated for Apex source classesand triggers each time one or more tests is run. Code coverage indicateshow many executable lines of code in classes and triggers have beentested by the test methods. Test methods are executed to generate codecoverage information. In one embodiment, code coverage can be specifiedas two lists of integers that includes covered lines and uncoveredlines. The term “code coverage percentage” refers to the number ofcovered lines divided by the sum of the number of covered lines and thenumber of uncovered lines. Additional details regarding the tooling APIcan be found in the Force.com Tooling API Reference and Developer Guide,Version 36.0, Spring '16, Last Updated: Mar. 3, 2016, which isincorporated by reference herein in its entirety.

The interaction between the user system 112 and various elements thatmake up the cloud-based computing platform 116 will be described greaterdetail below with reference to FIGS. 2 through 6.

FIG. 2 is a block diagram that shows modules 210, 240, 242, 244 executedby a user system 112 and their interaction with the test executionengine 150 and data store 124 via the SOQL interface 140 of FIG. 1 toexecute tests using grouped and filtered test classes in accordance withan embodiment. As noted above, the term “test class” refers to acollection of test method(s) that represent a test algorithm to validatethe accuracy and functionality of the application algorithm. In general,the test code pertaining to test method(s) or test classes call thesource code, to validate the returned output of source code withexpected output (e.g., assertion). These modules 210, 240, 242, 244 ofthe user system 112 include a test class filter module 210, a testmanagement module 239, and a test report generator 244.

The test class filter module 210 allows a user to programmaticallyselect and group a subset of tests that are to be executed at the testexecution engine 150 of the cloud-based computing environment 110 toincrease productivity during testing. Specifically, the test classfilter module 210 allows the user to group and then filter test classidentifiers to define the tests that are to be executed at the testexecution engine 150. The test class filter module 210 provides an easyway to run multiple, targeted tests at a time, which can in turnincrease productivity and efficiency of a user's testing efforts.

The test management module 239 can include a test enqueuer module 240, atest time tracker 241, and a test result poller/fetcher module 242. Thetest enqueuer module 240 receives a unique test class identifier arrayof filtered test class identifiers from the test class filter module 210and communicates test class identifiers (that identify the filtered testclasses) as a batch to the test execution engine 150 so that the testexecution engine 150 knows the batch of tests that are to be executed.The test time tracker 241 limits the time of each test and automaticallyaborts any tests that run for longer than a specified amount of time.For example, a user can input user-defined time limits for testexecution, and the test time tracker 241 can automatically stop any testthat exceeds those user-defined time limits. This way any tests thattake too long do not block others that are to be executed.

The test execution engine 150 relays a parent job identifier, used totrack test execution, to the test result poller/fetcher 242. The testresult poller/fetcher 242 uses a parent job identifier to monitor theprogress of the tests that are to be executed at the test executionengine 150 to determine whether all of the tests that are to be executedhave been completed. Upon determining that all of the tests that are tobe executed have been completed, the test result poller/fetcher 242fetches the test results by communicating the parent job identifier tothe SOQL interface 140, which retrieves the test results (correspondingto the parent job identifier) from data store 124 and provides the testresults to the test report generator 244. The test report generator 244generates a test report that includes the test execution results forfiltered set of test classes.

Tasks performed by the various elements in FIG. 2 will be described ingreater detail below with reference to FIG. 3, which is a flowchart thatillustrates an exemplary method 300 for grouping and filtering testclass identifiers and executing tests using the grouped and filteredtest class identifiers in accordance with an embodiment. FIG. 3 will bedescribed with reference to FIGS. 1-2. As a preliminary matter, itshould be understood that steps of the method 300 are not necessarilypresented in any particular order and that performance of some or allthe steps in an alternative order is possible and is contemplated. Thesteps have been presented in the demonstrated order for ease ofdescription and illustration. Further, steps can be added, omitted,and/or performed simultaneously without departing from the scope of theappended claims. It should also be understood that the illustratedmethod 300 can be stopped at any time. In certain embodiments, some orall steps of this process, and/or substantially equivalent steps, areperformed by execution of processor-readable instructions stored orincluded on a processor-readable medium, for example. For instance, inthe description of FIG. 3 that follows, the user system 112 and thecloud-based computing platform 116 will be described as performingvarious acts, tasks or steps, but it should be appreciated that thisrefers to processing system(s) of the user system 112 and the processorsystem 117 of the cloud-based computing platform 116 executinginstructions to perform those various acts, tasks or steps. Depending onthe implementation, the processor system 117 of the cloud-basedcomputing platform 116 can be centrally located, or distributed among anumber of server systems that work together. Furthermore, in thedescription of FIG. 3, it will be understood that any actions describedas being performed by the user are performed by the user interactingwith the user system 112.

Method 300 begins at 302, where a user of the user system 112 can inputcertain parameters including, for example, user credentials, manifestfilenames, regular expressions, test execution time limits, codecoverage thresholds, etc.

Users can populate class names in a manifest file and/or provide regularexpressions (regexes) to group and filter the Apex test classes.Multiple manifest files can be created for unit tests, functional tests,integration tests etc. As used herein, the term “manifest file” refersto a file that is populated by a user with a group or set of test classnames. The list/set/group of class names in each manifest file isstatically defined. Each manifest file has a name and includes a group(or set) of test class names for particular iteration of a test run, andis thus used to represent a group (or set) of class names. In otherwords, each manifest file has a name and a statically defined list oftest class names for each test class that is part of the group for thatmanifest file. Each of the test class names has a corresponding testclass identifier that identifies a test class to be executed. Eachmanifest file can be used to fetch a group (or set) of staticallydefined test classes identifiers.

The user of the user system can specify one or more of the regularexpressions as input parameters. As used herein, the term “regularexpression” refers to a pattern (e.g., group keys or key words such as astring value, word, group of words made from combination of any ASCIIcharacters). A regular expression can be used to perform a patternmatching or mapping to one or more test class identifiers associatedwith that regular expression, where each test class identifiercorresponds to a test class name of a test class. Each regularexpression can be used to fetch a particular group of test classidentifiers that correspond to test class names associated with thatregular expression. The pattern matching with a regular expression couldeither be a prefix, postfix or even a wild card match. Test classes canfollow a specific naming convention, that aligns with a regularexpression, and enables a target set of test classes, that areassociated with that regular expression, to be dynamically selected atruntime.

Inputting multiple manifest files and regular expressions can allow theuser to specify unique, targeted combinations of classes that are ofinterest to that user. This allows the user to consolidate differentgroups of classes to an uber group used to dynamically fetch the testclass identifiers during runtime. As such, the user does not have toknow and maintain a static list of test classes. The system isintelligent enough to build a test class set that matches the regularexpression(s) pattern specified by the user. This ability to build thelist of target test class names (or test class identifiers) can be usedto generate the target set of classes dynamically at runtime based onthe regular expression(s) defined by the user.

The user system 112 can include a validation module (not shown in FIG.2) that evaluates (at 304) the input parameters to determine whether ornot they are valid (e.g., to confirm that the user has entered validvalues for each of the required input parameters). When the validationmodule determines (at 304) that any of the input parameters are notvalid, it can generate an error message (at 305) and communicate theerror message to the user of the user system 112 via a user interface ofthe user system 112.

When the validation module determines that the input parameters arevalid, the method 300 proceeds to 306, where the user system 112initializes command line variables, and establishes (via a flowcontroller) a connection with the connection interface 130 of thecloud-based computing platform 116.

Once the user system 112 is communicatively coupled with the cloud-basedcomputing platform 116 via the connection interface 130, the user system112 can instantiate an instance of the test class filter module 210 thatis used to group and filter test class identifiers based on inputparameters specified by the user of the user system 112. As illustratedin FIG. 2, the test class filter module 210 interfaces with the SOQLinterface 140 of the cloud-based computing platform 116. The test classfilter module 210 includes a static test class fetcher 220, a dynamictest class fetcher 230, a unique class ID filter 236, and a logger 238.

The static test class fetcher 220 includes a manifest file processor 222and a manifest file reader and SOQL builder 224. The manifest fileprocessor 222 determines (at 308) whether manifest files werespecified/input by the user of the user system 112 when the user enteredthe input parameters. Multiple manifest files can be created for unittests, functional tests, integration tests, etc. When the manifest fileprocessor 222 determines (at 308) that manifest files were notspecified/input by the user of the user system 112, the method 300proceeds to 314. When the manifest file processor 222 determines (at308) that manifest files were specified/input by the user of the usersystem 112, the method 300 proceeds to 310, where the manifest fileprocessor 222 provides one or more manifest files to the manifest filereader and SOQL builder 224, and the manifest file reader and SOQLbuilder 224 processes each manifest file to read/extract test classnames, which the manifest file reader and SOQL builder 224 sends to theSOQL interface 140. Each manifest file has a name and includes a group(or set) of test class names for particular test run. Each manifest filecan be used to fetch a group (or set) of statically defined test classidentifiers.

The manifest file reader and SOQL builder 224 reads test class namesfrom each manifest file and sends test class names to the SOQL interface140. The SOQL interface 140 fetches corresponding test class identifiersfrom the data store 124 and provides the test class identifiers to themanifest file reader and SOQL builder 224. The manifest file reader andSOQL builder 224 combines the test class identifiers into an array thatis referred to herein as “a static test class identifier (ID) array.”The manifest file reader and SOQL builder 224 provides the static testclass identifier array to the dynamic test class fetcher 230. The statictest class identifier array provided from the static test class fetcher220 will ultimately be used, along with other test class identifiersgenerated by the dynamic test class fetcher 230, to generate a uniquetest class ID array of filtered test class identifiers.

The dynamic test class fetcher 230 includes a regular expressionprocessor 232 and a SOQL builder 234. The regular expression processor232 determines (at 314) whether regular expressions were specified/inputby the user of the user system 112 when the user entered the inputparameters.

When the regular expression processor 232 determines (at 314) thatregular expressions were not specified/input by the user of the usersystem 112, the method 300 proceeds to 320. However, if the regularexpression processor 232 determines (at 314) that one or more regularexpressions were specified/input by the user of the user system 112, themethod 300 proceeds to 316, where the regular expression processor 232provides one or more regular expressions to the SOQL builder 234.

The SOQL builder 234 processes each regular expression to generate anSOQL query with that regular expression, and sends the SOQL query withthat regular expression to the SOQL interface 140. At 318, for eachregular expression, the SOQL interface 140 contacts the data store 124and finds the matching test class names, and returns the correspondingtest class identifiers for the matching test class names (that wereretrieved from the data store 124) to the SOQL builder 234. The SOQLbuilder 234 combines the test class identifiers into an array that isreferred to herein as “a dynamic test class identifier array.” Thedynamic test class identifier array is then sent to the unique class IDfilter 236, along with the static test class identifier array that wasprovided from the static test class fetcher 220.

At 320, the unique class ID filter 236 uses the dynamic test classidentifier array and the static test class identifier array to generatea unique test class ID array of filtered test class identifiers that canbe used during further processing by the test execution engine 150. Inthe event that the unique class ID filter 236 determines that both thedynamic test class identifier array and the static test class identifierarray were empty, then it generates an error that is communicated to theuser via a user interface of the user system 112 instructing the user toinput a manifest file or regular expression to correct the problem sothat the unique test class ID array of filtered test class identifierscan be generated. The unique test class ID array of filtered test classidentifiers includes a subset of test class identifiers that have beengrouped or selected (via the static test class fetcher 220 and dynamictest class fetcher 230) based on user inputs and then filtered by theunique class ID filter 236. For example, in one embodiment, the uniqueclass ID filter 236 processes the two different test class ID arrays toidentify invalid class entries from either one of the test class IDarrays, and to identify any duplicate test class IDs that are in commonin the two class test class ID arrays. At 322, the logger 238 crates alog of any duplicates or any invalid class entries. For example, in oneembodiment, the unique class ID filter 236 combines the test classidentifiers from the static test class identifier array and the testclass identifiers from the dynamic test class identifier array into acombined set of test class identifiers, and then identifies any invalidclass IDs from either one of the test class ID arrays, and to identifiesany duplicate test class IDs that are in common in the two class testclass ID arrays. The unique class ID filter 236 can then eliminate theduplicate test class identifiers and the invalid test class identifiersfrom the combined set to generate the unique test class identifier arrayof filtered test class identifiers.

At 324, the unique class ID filter 236 can submit the unique test classidentifier array to a test enqueuer module 240, which provides theunique test class identifier array to the test execution engine 150. Thetest execution engine 150 uses the test class identifiers from theunique test class identifier array to determine which tests to execute.In other words, the test execution engine 150 uses this array offiltered test class identifiers to execute a particular subset of testsof all possible tests that could be executed. In response to receivingthe unique test class ID array, the test execution engine 150 provides aparent job identifier (ID) to the test result poller/fetcher 242.

The test result poller/fetcher 242 uses the parent job identifier totrack status of the test execution so that the test resultpoller/fetcher 242 can determine when all of the tests (corresponding tothe filtered test class identifiers specified in the unique test classidentifier array) have been completed. The test result poller/fetcher242 regularly polls the test execution engine 150 using the parent jobID to determine (at 326) whether all the tests associated with thisparent job ID have been executed/completed. When the test resultpoller/fetcher 242 determines (at 326) that all of the tests have notyet been executed by the test execution engine 150, the method 300 loopsat 326; however, in some embodiments, the loop at 326 can be limited bya timer or counter to prevent the loop from running indefinitely. Forexample, in one embodiment, the test time tracker 241 can maintain aclock that times how long each test has run for since the start ofexecution. When the test time tracker 241 determines that a test has runfor a time limit since the start of execution, then the test timetracker 241 can issue an instruction to terminate that test and/orgenerate/record an error message indicating that this test has failed.

When the test result poller/fetcher 242 determines (at 326) that all ofthe tests (corresponding to the filtered test class identifiersspecified in the unique test class identifier array) have been executedby the test execution engine 150, the test result poller/fetcher 242provides the parent job identifier to the SOQL interface 140 as part ofa request for the test results. The SOQL interface 140 can then retrievethe test results from the data store 124, and provide the retrieved testresults to a test report generator 244. The test report generator 244then generates a test report at 328 based on the test results.

FIG. 4 is a block diagram that shows other modules 250, 280, 282, 284,286 executed by the user system 112 and their interaction with the SOQLinterface 140, tooling API 160, code coverage computation unit 170 anddata store 124 of FIG. 1 to generate code coverage metrics using groupedand filtered source class identifiers in accordance with an embodiment.As noted above, the term “source class” refers to a unit of source codeof an application. Modules 250, 280, 282, 284, 286 of the user system112 include a source class filter module 250, a tooling API invoker 280,an OAuth generator 282, a code coverage report generator 284, and aquality metrics verification module 286.

The source class filter module 250 allows a user to programmaticallyselect a subset of source class identifiers that are to be provided tothe code coverage computation unit 170 and used during code coveragecomputations to improve precision of code coverage analysis.Specifically the source class filter module 250 allows a user to groupand filter source class identifiers that specify which source classes“targeted” code coverage metrics are to be computed for, and providesthe user-selected subset of source class identifiers to the codecoverage computation unit 170 (via the tooling API interface 160) foruse during code coverage computations. In essence this allows the userto select (via input parameters specified by the user) a subset of thecode base that code coverage metrics are to be computed for. By allowingthe user to group and filter the source classes for which the user wantsto exercise code coverage computations, the source class filter module250 provides an easy way to specify highly-targeted sets of sourceclasses to be used during code coverage computations, which can in turnincrease productivity and efficiency of the user's code coverageanalysis and testing.

The tooling API invoker 280 communicates with the OAuth generator 282 toobtain an organization token from the OAuth generator 282 for theparticular session. OAuth is an open standard for authorization thatprovides to clients a secure delegated access to server resources onbehalf of a resource owner. OAuth standards specify a process forresource owners to authorize access to their server resources withoutsharing their credentials. OAuth allows access tokens to be issued toclients by an authorization server, with the approval of the resourceowner. The client can then use the access token to access the protectedresources hosted by the resource server. The tooling API invoker 280provides the organization token to the tooling API 160 to establish asession with the tooling API 160. The tooling API invoker 280 receives aunique source class identifier (ID) array of filtered source classidentifiers are specified by the user using manifest files and/orregular expressions, and provides the filtered source class identifiers(from the unique source class identifier array) to the tooling API 160.

The tooling API 160 interfaces with a code coverage computation unit 170that computes code coverage metrics for the source classes thatcorrespond to the filtered set of source class identifiers. The sourceclasses, that correspond to the filtered source class identifiersspecified in the unique source class identifier (ID) array, correspondto a subset of the code base that the user specifies by inputting acombination of manifest file(s) and/or regular expression(s). The codecoverage metrics specify the amount of source code that is covered bycertain tests and shows how thoroughly the chosen subset of code hasbeen tested. The code coverage computation unit 170 stores the codecoverage metrics in the data store 124. The tooling API 160 can retrieveand provide the code coverage metrics for the filtered set of sourceclasses to the code coverage report generator 284.

The code coverage report generator 284 uses the code coverage metricsfor the source classes to generate an HTML report. The report includesall of the code coverage metrics for the filtered set of source classesconsolidated into an actionable report. The user can control thegranularity of the report (in terms of how the code coverage metrics areconsolidated) by specifying code coverage results down to complexmodules (and even line numbers) for developers.

Processes performed by the various elements in FIG. 4 will be describedin greater detail below with reference to FIGS. 5A and 5B, whichcollectively illustrate a flowchart that illustrates an exemplary method500 for grouping and filtering source class identifiers and generatingcode coverage metrics using grouped and filtered source classidentifiers in accordance with an embodiment. FIGS. 5A and 5B will bedescribed with reference to FIGS. 1-4. As a preliminary matter, itshould be understood that steps of the method 500 are not necessarilypresented in any particular order and that performance of some or allthe steps in an alternative order is possible and is contemplated. Thesteps have been presented in the demonstrated order for ease ofdescription and illustration. Further, steps can be added, omitted,and/or performed simultaneously without departing from the scope of theappended claims. It should also be understood that the illustratedmethod 500 can be stopped at any time. In certain embodiments, some orall steps of this process, and/or substantially equivalent steps, areperformed by execution of processor-readable instructions stored orincluded on a processor-readable medium, for example. For instance, inthe description of FIGS. 5A and 5B that follows, the user system 112 andelements of the cloud-based computing platform 116 will be described asperforming various acts, tasks or steps, but it should be appreciatedthat this refers to processing system(s) of the user system 112 and theprocessor system 117 of the cloud-based computing platform 116 executinginstructions to perform those various acts, tasks or steps. Depending onthe implementation, the processor system 117 of the cloud-basedcomputing platform 116 can be centrally located, or distributed among anumber of systems that work together. Furthermore, in the description ofFIGS. 5A and 5B, it will be understood that any actions described asbeing performed by the user are performed by the user interacting withthe user system 112.

Method 500 begins at 502, where a user of the user system 112 can inputcertain parameters including, for example, user credentials, manifestfilenames, regular expressions, test execution time limits, codecoverage thresholds, etc.

The user system 112 can include a validation module (not shown in FIG.4) that evaluates (at 504) the input parameters to determine whether ornot they are valid (e.g., to confirm that the user has entered validvalues for each of the required input parameters). When the validationmodule determines (at 504) that any of the input parameters are notvalid, it can generate an error message (at 505) and communicate theerror message to the user of the user system 112 via a user interface ofthe user system 112.

When the validation module determines that the input parameters arevalid, the method 500 proceeds to 506, where the user system 112initializes command line variables, and establishes (via a flowcontroller) a connection with the connection interface 130 of thecloud-based computing platform 116.

Once the user system 112 is communicatively coupled with the cloud-basedcomputing platform 116 via the connection interface 130, at 506, theuser system 112, can instantiate an instance of the source class filtermodule 250 that interfaces with the SOQL interface 140 of thecloud-based computing platform 116. As illustrated in FIG. 4, the sourceclass filter module 250 includes a static source class fetcher 260, adynamic source class fetcher 270, a unique class ID filter 276, and alogger 278.

The static source class fetcher 260 includes a manifest file processor262 and a manifest file reader and SOQL builder 264. The manifest fileprocessor 262 determines (at 508) whether manifest files werespecified/input by the user of the user system 112 when the user enteredthe input parameters. When the manifest file processor 262 determines(at 508) that manifest files were not specified/input by the user of theuser system 112, the method 500 proceeds to 514.

When the manifest file processor 262 determines (at 508) that manifestfiles were specified/input by the user of the user system 112, themethod 500 proceeds to 510, where the manifest file processor 262provides one or more manifest files to the manifest file reader and SOQLbuilder 264, and the manifest file reader and SOQL builder 264 processeseach manifest file to read/extract source class names specified in thatmanifest file, and sends the source class names to the SOQL interface140. Each manifest file has a name and includes a group (or set) ofsource class names for each source class that is part of that manifestfile. Each manifest file can be used to fetch a group (or set) ofstatically defined source class identifiers.

The SOQL interface 140 uses the source class names to fetchcorresponding source class identifiers (IDs) from the data store 124 andprovides the retrieved source class identifiers to the manifest filereader and SOQL builder 264. The manifest file reader and SOQL builder264 combines the source class identifiers into an array that is referredto herein as “a static source class identifier array.”

The manifest file reader and SOQL builder 264 provides the static sourceclass identifier array to the dynamic source class fetcher 270. Thestatic source class identifier array provided from the static sourceclass fetcher 260 can then be used along with other dynamic source classidentifiers (described below) generated by the dynamic source classfetcher 270 to generate a unique source class ID array of filteredsource class identifiers.

The dynamic source class fetcher 270 includes a regular expressionprocessor 272 and a SOQL builder 274. The regular expression processor272 determines (at 514) whether one or more regular expressions werespecified/input by the user of the user system 112 when the user enteredthe input parameters. The user can input more than one such regularexpressions (as input parameters) to consolidate different groups ofsource classes to an uber group that can be used to dynamically fetchthe source class identifiers during runtime. Each regular expression isa pattern (e.g., group keys or key words such as a string value, word,group of words made from ASCII characters) that can be processed toperform a pattern matching or mapping to one or more source class namesthat correspond a particular group of source classes. The power thatthis capability provides on top of manifest file approach is that userdoes not have to know and maintain the static list of source classes.The system is intelligent enough to build the source class set thatmatch the regular expression(s) specified by the user. When the regularexpression processor 272 determines (at 514) that regular expressionswere not specified/input by the user of the user system 112, the method500 proceeds to 520.

When the regular expression processor 272 determines (at 514) thatregular expressions were specified/input by the user of the user system112, the method 500 proceeds to 516, where the regular expressionprocessor 272 provides one or more regular expressions to the SOQLbuilder 274. The SOQL builder 274 processes each regular expression togenerate an SOQL query with that regular expression, and sends the SOQLquery with that regular expression to the SOQL interface 140.

At 518, for each regular expression, the SOQL interface 140 contacts thedata store 124 and finds the matching source class names, and returnsthe corresponding source class identifiers for the matching source classnames (that were retrieved from the data store 124) to the SOQL builder274. The SOQL builder 274 combines the source class identifiers into anarray that is referred to herein as “a dynamic source class identifierarray.” The dynamic source class identifier array is then sent to theunique class ID filter 276, along with the static source classidentifier array that was provided from the static source class fetcher260.

At 520, the unique class ID filter 276 uses the dynamic source classidentifier array and the static source class identifier array togenerate a unique source class identifier (ID) array of filtered sourceclass identifiers that can be used during further processing during acode coverage computation. For example, in one embodiment, the uniqueclass ID filter 276 combines the source class identifiers from thestatic source class identifier array and the source class identifiersfrom the dynamic source class identifier array into a combined set ofsource class identifiers, and then identifies any invalid class entriesfrom either one of the source class ID arrays, and identifies anyduplicate source class IDs that are in common in the two class sourceclass ID arrays. The unique class ID filter 276 can then eliminate theduplicate source class identifiers and the invalid source classidentifiers from the combined set to generate the unique source classidentifier array of filtered source class identifiers. At 522, thelogger 278 creates a log of any duplicates or any invalid class entries.

The unique class ID filter 276 can submit the unique source classidentifier array to a tooling API invoker 280 via the OAuth generator282. The tooling API invoker 280 requests an organization token from theOAuth generator 282, and the OAuth generator 282 provides theorganization token to the tooling API invoker 280. The tooling APIinvoker 280 then sends the organization token and the unique sourceclass identifier array to the tooling API 160. The tooling API 160 andthe tooling API invoker 280 of the user system 112 use the organizationtoken to establish a session between the tooling API invoker 280 and thetooling API 160. In other words, the organization token authorizes thetooling API invoker 280 to communicate with the tooling API 160.

At 524, the tooling API 160 provides the source class identifiers fromthe unique source class identifier array to the code coveragecomputation unit 170, and the code coverage computation unit 170 usessource class identifiers from the unique source class identifier arrayto compute targeted code coverage metrics. In one implementation, thecode coverage computation unit 170 can also compute organization widecode coverage metrics for all source class IDs. The “organization widecode coverage metrics” are code coverage metrics that indicate codecoverage for the entire organization, whereas the “targeted codecoverage metrics” are code coverage metrics that are specific to onlythe filtered source classes specified via the source class identifiersthat are specified in the unique source class ID array. For instance,the targeted code coverage metrics can be a subset of code coveragemetrics that are of interest to a specific group such as a specificsub-tenant or group of sub-tenants, such as a specific business unit orfunctional group. The code coverage computation unit 170 can store thecomputed organization wide code coverage metrics and targeted codecoverage metrics in the data store 124. The tooling API 160 can receivethe code coverage metrics from the data store 124, and then provide theorganization wide code coverage metrics and targeted code coveragemetrics to the code coverage report generator 284.

At 526, the code coverage report generator 284 then generates aconsolidated report that includes the various code coverage metrics, andoptionally test results that were generated (as described above) by thetest report generator 244 of FIG. 2.

FIG. 6 is a screenshot of an example of a consolidated report that canbe displayed at a user interface of the user system 112 in accordancewith one exemplary implementation of an embodiment. In this embodiment,the consolidated report can include test results from the test reportand the code coverage metrics. The consolidated “ApexUnit” report caninclude a code coverage summary 610 indicating what the computed teamcode coverage was determined to be along with the customized team codecoverage threshold, and what the computed organization wide codecoverage was determined to be along with the customized organizationwide code coverage threshold. The consolidated report can also include atest execution summary 620 that indicates how many total test classeswere executed during testing, how many total test methods were executedduring testing, how many total test methods passed during testing, andhow many total test methods failed during testing. The consolidatedreport can also include an Apex Test Report 630 that includes ahyperlink to a Detailed Test Report (not shown) that includes testresults. Upon navigating to the hyperlink, it renders detailed testexecution metrics with high level summary of how many test passed andhow many failed. It provides a drill down scenario to dig deep into thefailed test cases citing related logs and information of specificfailure in a drill down fashion. The consolidated report can alsoinclude a Detailed Code Coverage report 640. The Detailed Code Coveragereport includes a table 640 with columns for Apex Class Name, APIVersion, Percent Code Coverage, number of covered lines, number ofuncovered lines, a list of covered lines, a list of uncovered lines, andlength without comments (Bytes). The Detailed Code Coverage report alsoincludes a list 650 of duplicate Apex class names across manifest filesand regular expressions. The Detailed Code Coverage report 640 alsoincludes a list 660 of invalid (or non-existent) Apex class names acrossmanifest files and regular expressions.

Referring again to FIG. 5B, at 528, the quality metrics verificationmodule 286 can determine whether quality metrics or measures have beenverified. To determine whether quality measures are verified, thequality metrics verification module can validate test results and codecoverage metrics. In one embodiment, validation of test results refersto determining if any of the tests that were to be executed by the testexecution engine 150 have failed (e.g., not completed). If the qualitymetrics verification module determines that any of the tests that wereto be executed by the test execution engine 150 failed, the qualitymetrics verification module 286 generates an error message andcommunicates the error message to the user of the user system 112 via auser interface of the user system 112. If all of the tests that were tobe executed by the test execution engine 150 have successfullycompleted, then the quality metrics verification module 286 validates tothe code coverage results by determining if whether code coverage isgreater than or equal to a user-specified code coverage threshold (e.g.,75% in one implementation).

When the quality metrics verification module 286 determines, based onthe code coverage metrics, that code coverage is greater than or equalto a user-specified code coverage threshold, then the code coverage isdetermined to be valid. When the quality metrics verification module 286determines, based on the code coverage metrics, that code coverage isless than the user-specified code coverage threshold, then the codecoverage is determined to be invalid and the quality metricsverification module 286 generates an error message and communicates theerror message to the user of the user system 112 via a user interface ofthe user system 112. Thus, when the quality metrics verification module286 determines that the test results or the code coverage metrics areunacceptable, the method 500 proceeds to 530. At 530, the qualitymetrics verification module 286 can generate an error message andcommunicate the error message to the user of the user system 112 via auser interface of the user system 112. When the quality metricsverification module 286 determines that both the test results and thecode coverage metrics are acceptable, the method proceeds to 532, wherethe method 500 ends.

FIG. 7 is a schematic block diagram of a user system in accordance withan embodiment. FIG. 7 will be described with reference to FIGS. 1-5. Theuser system 112 can include one or more processing system(s) 702, mainmemory 704, a network interface device (NID) 710, a chipset 712, inputsystems 716, and output systems 718. It will be appreciated that theuser system 112 may not include all of the components shown in FIG. 7,may include other components that are not explicitly shown in FIG. 7, ormay utilize an architecture completely different than that shown in FIG.7.

The chipset 712 is usually located on a motherboard of the user system112. The chipset 712 is a set of electronic components (e.g., in anintegrated circuit) that interconnects and manages the data flow betweenthe processing system(s) 702 and other elements of the user system 112and any peripherals that are connected to the user system 112. Forinstance, the chipset 712 provides an interface between the processingsystem(s) 702 and the main memory 704, and also includes functionalityfor providing network connectivity through the NID 710, such as agigabit Ethernet adapter. The chipset 712 typically contains theprocessor bus interface (also known as a front-side bus), memorycontrollers, bus controllers, I/O controllers, etc.

Processing system(s) 702 represents one or more general-purposeprocessing devices such as a microprocessor, central processing unit, orthe like. More particularly, the processing system(s) 702 may be acomplex instruction set computing (CISC) microprocessor, reducedinstruction set computing (RISC) microprocessor, very long instructionword (VLIW) microprocessor, or a processor implementing otherinstruction sets or processors implementing a combination of instructionsets. The processing system(s) 702 may also be one or morespecial-purpose processing devices such as an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA), adigital signal processor (DSP), network processor, or the like.

The processing system(s) 702 can include one or more central processingunits (“CPUs”) that operate in conjunction with the chipset 712. Theprocessing system(s) 702 perform arithmetic and logical operationsnecessary for the operation of the user system 112. The processingsystem(s) 702 can perform the necessary operations by transitioning fromone discrete, physical state to the next through the manipulation ofswitching elements that differentiate between and change these states.Switching elements may generally include electronic circuits thatmaintain one of two binary states, such as flip-flops, and electroniccircuits that provide an output state based on the logical combinationof the states of one or more other switching elements, such as logicgates. These basic switching elements may be combined to create morecomplex logic circuits, including registers, adders-subtractors,arithmetic logic units, floating-point units, and the like.

The NID 710 is capable of connecting the user system 112 to othercomputers over the network 114. The network 114 can be an Ethernet orGigabyte Ethernet LAN, a fiber ring, a fiber star, wireless, optical,satellite, a WAN, a MAN, or any other network technology, topology,protocol, or combination thereof.

Input system(s) 716 (or input device(s)) allow the user to inputinformation to the user system and can include things such as akeyboard, a mouse or other cursor pointing device, a pen, a voice inputdevice, a touch input device, a webcam device, a microphone, etc. Outputsystem(s) 718 (or output device(s)) present information to the user ofthe user system and can include things such as a display, monitor,speakers, or the like. All of these systems/devices are well known inthe art and need not be discussed at length here.

The chipset 712 can provide an interface to various forms ofcomputer-readable storage media including a main memory 704 (e.g.,read-only memory (ROM), flash memory, dynamic random access memory(DRAM) such as synchronous DRAM (SDRAM), a hard disk, etc.). Theprocessing system(s) 702 can communicate with the various forms forcomputer-readable storage media via the chipset 712 and appropriatebuses.

The main memory 704 may be composed of many different types of memorycomponents. The main memory 704 can include non-volatile memory (such asread-only memory (ROM) 706, flash memory, etc.), volatile memory (suchas random access memory (RAM) 708), or some combination of the two. TheRAM 708 can be any type of suitable random access memory including thevarious types of dynamic random access memory (DRAM) such as SDRAM, thevarious types of static RAM (SRAM). The main memory 704 (as well as theprocessing system(s) 702) may be distributed throughout the user system112.

The ROM 706 of the main memory 704 can be used to store firmware thatincludes program code containing the basic routines that help to startup the user system 112 and to transfer information between elementswithin the user system 112. The ROM of the main memory 704 may alsostore other software components necessary for the operation of the usersystem 112.

The RAM 708 stores programs/instructions 730 or executable code for oneor more programs that can be loaded and executed at processing system(s)702 to perform various functions. The programs/instructions 730 arecomputer readable program code that can be stored in RAM 708 (or other anon-transitory computer readable medium of the user system 112) that canbe read and executed by processing system(s) 702 to perform variousacts, tasks, functions, and steps as described herein. Examples ofprograms/instructions 730 that are stored in the RAM 708 include a testclass filter module 210, a test enqueuer module 240, a test resultpoller/fetcher 242, a test report generator 244, a source class filtermodule 250, a tooling API invoker 280, an OAuth generator 282, a codecoverage report generator 284, and a quality metrics verification module286 in accordance with the embodiments described herein.

Various functions performed by the processing system(s) 702 upon loadingand executing the test class filter module 210, the test enqueuer module240, the test result poller/fetcher 242, the test report generator 244,the source class filter module 250, the tooling API invoker 280, anOAuth generator 282, the code coverage report generator 284, and thequality metrics verification module 286 have been described above withreference to FIGS. 1-6 and will not be repeated here for sake ofbrevity.

FIG. 8 illustrates a block diagram of an example environment 810 inwhich the embodiments described herein may be implemented. Environment810 may include user systems 812, network 814, system 816, processorsystem 817, application platform 818, network interface 820, tenant datastorage 822, system data storage 824, program code 826, and processspace 828. In other embodiments, environment 810 may not have all of thecomponents listed and/or may have other elements instead of, or inaddition to, those listed above.

Environment 810 is an environment in which an on-demand database serviceexists. User system 812 may be any machine or system that is used by auser to access a database system. For example, any of user systems 812can be a handheld computing device, a mobile phone, a laptop computer, awork station, and/or a network of computing devices. As illustrated inFIG. 8 (and in more detail in FIG. 9) user systems 812 might interactvia a network 814 with an on-demand database service, which is system816. System 816 may also be referred to as a cloud service provider.System 816 provides its resources to customers (e.g., end users) as aservice.

An on-demand database service, such as system 816, is a database systemthat is made available to outside users who do not need to necessarilybe concerned with building and/or maintaining the database system, butinstead may be available for more general use when the users need thedatabase system (e.g., on the demand of the users). Some on-demanddatabase services may store information from one or more tenants storedinto tables of a common database image to form a multi-tenant databasesystem (MTS). Accordingly, “on-demand database service 816” and “system816” will be used interchangeably herein. A database image may includeone or more database objects. A relational database management system(RDMS) or the equivalent may execute storage and retrieval ofinformation against the database object(s). Application platform 818 maybe a framework that allows the applications of system 816 to run, suchas the hardware and/or software, e.g., the operating system. In anembodiment, system 816 may include an application platform 818 thatenables creating, managing, and executing one or more applicationsdeveloped for an on-demand database service, for users accessing theon-demand database service via user systems 812, or for third partyapplication developers accessing the on-demand database service via usersystems 812.

The users of user systems 812 may differ in their respective capacities,and the capacity of a particular user system 812 might be entirelydetermined by permissions (permission levels) for the current user. Forexample, where a salesperson is using a particular user system 812 tointeract with system 816, that user system has the capacities allottedto that salesperson. However, while an administrator is using that usersystem to interact with system 816, that user system has the capacitiesallotted to that administrator. In systems with a hierarchical rolemodel, users at one permission level may have access to applications,data, and database information accessible by a lower permission leveluser, but may not have access to certain applications, databaseinformation, and data accessible by a user at a higher permission level.Thus, different users will have different capabilities with regard toaccessing and modifying application and database information, dependingon a user's security or permission level.

Network 814 is any network or combination of networks of devices thatcommunicate with one another. For example, network 814 can be any one orany combination of a local area network (LAN), wide area network (WAN),telephone network, wireless network, point-to-point network, starnetwork, token ring network, hub network, or other appropriateconfiguration. As the most common type of computer network in currentuse is a transfer control protocol and Internet protocol (TCP/IP)network, such as the global internetwork of networks often referred toas the “Internet” with a capital “I.” That network will be used in manyof the examples herein. However, it should be understood that thenetworks used with the embodiment described herein use are not solimited, although TCP/IP is a frequently implemented protocol.

User systems 812 might communicate with system 816 using TCP/IP and, ata higher network level, use other common Internet protocols tocommunicate, such as hypertext transfer protocol (HTTP), file transferprotocol (FTP), Andrew file system (AFS), wireless application protocol(WAP), etc. In an example where HTTP is used, user system 812 mightinclude an HTTP client commonly referred to as a “browser” for sendingand receiving HTTP messages to and from an HTTP server at system 816.Such an HTTP server might be implemented as the sole network interfacebetween system 816 and network 814, but other techniques might be usedas well or instead. In some implementations, the interface betweensystem 816 and network 814 includes load sharing functionality, such asround-robin HTTP request distributors to balance loads and distributeincoming HTTP requests evenly over a plurality of servers. At least asfor the users that are accessing that server, each of the plurality ofservers has access to the MTS' data; however, other alternativeconfigurations may be used instead.

In one embodiment, system 816, shown in FIG. 8, implements a web-basedcustomer relationship management (CRM) system. For example, in oneembodiment, system 816 includes application servers configured toimplement and execute CRM software applications as well as to providerelated data, code, forms, webpages and other information to and fromuser systems 812. The application servers are also configured to storeto, and retrieve from, a database system related data, objects, andWebpage content. With a multi-tenant system, data for multiple tenantsmay be stored in the same physical database object. Tenant data may bearranged such that data of one tenant is kept logically separate fromthat of other tenants so that one tenant does not have access to anothertenant's data, unless such data is expressly shared. In certainembodiments, system 816 implements applications other than, or inaddition to, a CRM application. For example, system 816 may providetenant access to multiple hosted (standard and custom) applications,including a CRM application. User (or third party application developer)software applications, which may or may not include CRM, may besupported by the application platform 818, which manages the creationand storage of the applications into one or more database objects, andexecuting of the applications in a virtual machine in the process space828 of the system 816. The terms “application,” “software application,”“software package,” “software code,” and “program code” are usedinterchangeably.

One arrangement for elements of system 816 is shown in FIG. 8, includinga network interface 820, application platform 818, tenant data storage822 for tenant data 823, system data storage 824 for system data 825accessible to system 816 and possibly multiple tenants, program code 826for implementing various functions of system 816, and a process space828 for executing MTS system processes and tenant-specific processes,such as running applications as part of an application hosting service.Additional processes that may execute on system 816 include databaseindexing processes.

Several elements in the system shown in FIG. 8 include conventional,well-known elements that are explained only briefly here. For example,each user system 812 could include a desktop personal computer,workstation, laptop, PDA, cell phone, or any wireless access protocol(WAP) enabled device or any other computing device capable ofinterfacing directly or indirectly to the Internet or other networkconnection. User system 812 typically runs an HTTP client, e.g., abrowsing program, such as Google's Chrome browser, Microsoft's InternetExplorer browser, Netscape's Navigator browser, Opera's browser, or aWAP-enabled browser in the case of a cell phone, PDA or other wirelessdevice, or the like, allowing a user (e.g., subscriber of themulti-tenant database system) of user system 812 to access, process andview information, pages and applications available to it from system 816over network 814. Each user system 812 also typically includes one ormore user interface devices, such as a keyboard, a mouse, trackball,touch pad, touch screen, pen or the like, for interacting with agraphical user interface (GUI) provided by the browser on a display(e.g., a monitor screen, liquid crystal display (LCD) monitor, etc.) inconjunction with pages, forms, applications and other informationprovided by system 816 or other systems or servers. For example, theuser interface device can be used to access data and applications hostedby system 816, and to perform searches on stored data, and otherwiseallow a user to interact with various GUI pages that may be presented toa user. As discussed above, embodiments are suitable for use with theInternet, which refers to a specific global internetwork of networks.However, it should be understood that other networks can be used insteadof the Internet, such as an intranet, an extranet, a virtual privatenetwork (VPN), a non-TCP/IP based network, any LAN or WAN or the like.

According to one embodiment, each user system 812 and all of itscomponents are operator configurable using applications, such as abrowser, including computer code run using a central processing unitsuch as an Intel Pentium® processor or the like. Similarly, system 816(and additional instances of an MTS, where more than one is present) andall of their components might be operator configurable usingapplication(s) including computer code to run using a central processingunit such as processor system 817, which may include an Intel Pentium®processor or the like, and/or multiple processor units. A computerprogram product embodiment includes a machine-readable storage medium(media) having instructions stored thereon/in which can be used toprogram a computer to perform any of the processes of the embodimentsdescribed herein. Computer code for operating and configuring system 816to intercommunicate and to process webpages, applications and other dataand media content as described herein are preferably downloaded andstored on a hard disk, but the entire program code, or portions thereof,may also be stored in any other volatile or non-volatile memory mediumor device as is well known, such as a read-only memory (ROM) orrandom-access memory (RAM), or provided on any media capable of storingprogram code, such as any type of rotating media including floppy disks,optical discs, digital versatile disk (DVD), compact disk (CD),microdrive, and magneto-optical disks, and magnetic or optical cards,nanosystems (including molecular memory integrated circuits (ICs)), orany type of media or device suitable for storing instructions and/ordata. Additionally, the entire program code, or portions thereof, may betransmitted and downloaded from a software source over a transmissionmedium, e.g., over the Internet, or from another server, as is wellknown, or transmitted over any other conventional network connection asis well known (e.g., extranet, virtual private network (VPN), LAN, etc.)using any communication medium and protocols (e.g., TCP/IP, HTTP, HTTPS,Ethernet, etc.) as are well known. It will also be appreciated thatcomputer code for implementing embodiments can be implemented in anyprogramming language that can be executed on a client system and/orserver or server system such as, for example, C, C++, HTML, any othermarkup language, Java™, JavaScript, ActiveX, any other scriptinglanguage, such as VBScript, and many other programming languages as arewell known may be used. (Java™ is a trademark of Sun Microsystems,Inc.).

According to one embodiment, each system 816 is configured to providewebpages, forms, applications, data and media content to user (client)systems 812 to support the access by user systems 812 as tenants ofsystem 816. As such, system 816 provides security mechanisms to keepeach tenant's data separate unless the data is shared. If more than oneMTS is used, they may be located in close proximity to one another(e.g., in a server farm located in a single building or campus), or theymay be distributed at locations remote from one another (e.g., one ormore servers located in city A and one or more servers located in cityB). As used herein, each MTS could include one or more logically and/orphysically connected servers distributed locally or across one or moregeographic locations. Additionally, the term “server” is meant toinclude a computer system, including processing hardware and processspace(s), and an associated storage system and database application(e.g., object oriented database management system (OODBMS) or rationaldatabase management system (RDBMS)) as is well known in the art. Itshould also be understood that “server system” and “server” are oftenused interchangeably herein. Similarly, the database object describedherein can be implemented as single databases, a distributed database, acollection of distributed databases, a database with redundant online oroffline backups or other redundancies, etc., and might include adistributed database or storage network and associated processingintelligence.

FIG. 9 illustrates a block diagram of another example environment 810,which may be used to implement the embodiments described herein. Some ofthe elements in FIG. 9 overlap with those in FIG. 8, and therefore FIG.9 will be described with reference to FIG. 8, and common numbering willbe used for elements in FIG. 9 that are shown in FIG. 8. FIG. 9 alsoillustrates elements of system 816 and various interconnections,according to one embodiment. FIG. 9 shows that user system 812 mayinclude processor system 812A (analogous to processing system(s) 702 inFIG. 7), memory system 812B (analogous to main memory 704 in FIG. 7),input system 812C (analogous to input system(s) 716 in FIG. 7), andoutput system 812D (analogous to output system(s) 718 in FIG. 7). FIG. 9shows network 814 and system 816. FIG. 9 also shows that system 816 mayinclude tenant data storage 822, tenant data 823, system data storage824, system data 825, user interface (UI) 930, application programinterface (API) 932, PL/Salesforce.com object query language (PL/SOQL)934, save routines 936, application setup mechanism 938, applicationsservers 900 ₁-900 _(N), system process space 902, tenant process spaces904, tenant management process space 910, tenant storage area 912, userstorage for tenant data 914, and application metadata 916. In otherembodiments, environment 810 may not have the same elements as thoselisted above and/or may have other elements instead of, or in additionto, those listed above.

User system 812, network 814, system 816, tenant data storage 822, andsystem data storage 824 were discussed above in FIG. 8. Regarding usersystem 812, processor system 812A may be any combination of one or moreprocessors. Memory system 812B may be any combination of one or morememory devices, short term, and/or long term memory. Input system 812Cmay be any combination of input devices, such as one or more keyboards,mice, trackballs, scanners, cameras, and/or interfaces to networks.Output system 812D may be any combination of output devices, such as oneor more monitors, printers, and/or interfaces to networks. As shown inFIG. 8, system 816 may include a network interface 820 (of FIG. 8)implemented as a set of HTTP application servers 900, an applicationplatform 818, tenant data storage 822, and system data storage 824. Alsoshown is system process space 902, including individual tenant processspaces 904 and a tenant management process space 910. Each applicationserver 900 may be configured to access tenant data storage 822 and thetenant data 823 therein, and system data storage 824 and the system data825 therein to serve requests of user systems 812. The tenant data 823might be divided into individual tenant storage areas 912, which can beeither a physical arrangement and/or a logical arrangement of data.Within each tenant storage area 912, user storage 914 and applicationmetadata 916 might be similarly allocated for each user. For example, acopy of a user's most recently used (MRU) items might be stored to userstorage 914. Similarly, a copy of MRU items for an entire organizationthat is a tenant might be stored to tenant storage area 912. A UI 930provides a user interface and an API 932 provides an applicationprogrammer interface to system 816 resident processes and to usersand/or developers at user systems 812. The tenant data and the systemdata may be stored in various databases, such as one or more Oracle™databases.

Application platform 818 includes an application setup mechanism 938that supports application developers' creation and management ofapplications, which may be saved as metadata into tenant data storage822 by save routines 936 for execution by subscribers as one or moretenant process spaces 904 managed by tenant management process 910, forexample. Invocations to such applications may be coded using PL/SOQL 934that provides a programming language style interface extension to API932. Invocations to applications may be detected by one or more systemprocesses, which manage retrieving application metadata 916 for thesubscriber, making the invocation and executing the metadata as anapplication in a virtual machine.

Each application server 900 may be communicably coupled to databasesystems, e.g., having access to system data 825 and tenant data 823, viaa different network connection. For example, one application server 900₁ might be coupled via the network 814 (e.g., the Internet), anotherapplication server 900 _(N-1) might be coupled via a direct networklink, and another application server 900 _(N) might be coupled by yet adifferent network connection. Transfer control protocol and Internetprotocol (TCP/IP) are typical protocols for communicating betweenapplication servers 900 and the database system. However, it will beapparent to one skilled in the art that other transport protocols may beused to optimize the system depending on the network connection used.

In certain embodiments, each application server 900 is configured tohandle requests for any user associated with any organization that is atenant. Because it is desirable to be able to add and remove applicationservers from the server pool at any time for any reason, there ispreferably no server affinity for a user and/or organization to aspecific application server 900. In one embodiment, therefore, aninterface system implementing a load balancing function (e.g., an F5Big-IP load balancer) is communicably coupled between the applicationservers 900 and the user systems 812 to distribute requests to theapplication servers 900. In one embodiment, the load balancer uses aleast connections algorithm to route user requests to the applicationservers 900. Other examples of load balancing algorithms, such as roundrobin and observed response time, also can be used. For example, incertain embodiments, three consecutive requests from the same user couldhit three different application servers 900, and three requests fromdifferent users could hit the same application server 900. In thismanner, system 816 is multi-tenant, wherein system 816 handles thestorage of, and access to, different objects, data and applicationsacross disparate users and organizations.

As an example of storage, one tenant might be a company that employs asales force where each salesperson uses system 816 to manage his or hersales process. Thus, a user might maintain contact data, leads data,customer follow-up data, performance data, goals and progress data,etc., all applicable to that user's personal sales process (e.g., intenant data storage 822). In an example of an MTS arrangement, since allof the data and the applications to access, view, modify, report,transmit, calculate, etc., can be maintained and accessed by a usersystem having nothing more than network access, the user can manage hisor her sales efforts and cycles from any of many different user systems.For example, if a salesperson is visiting a customer and the customerhas Internet access in their lobby, the salesperson can obtain criticalupdates as to that customer while waiting for the customer to arrive inthe lobby.

While each user's data might be separate from other users' dataregardless of the employers of each user, some data might beorganization-wide data shared or accessible by a plurality of users orall of the users for a given organization that is a tenant. Thus, theremight be some data structures managed by system 816 that are allocatedat the tenant level while other data structures might be managed at theuser level. Because an MTS might support multiple tenants includingpossible competitors, the MTS should have security protocols that keepdata, applications, and application use separate. Also, because manytenants may opt for access to an MTS rather than maintain their ownsystem, redundancy, up-time, and backup are additional functions thatmay be implemented in the MTS. In addition to user-specific data andtenant specific data, system 816 might also maintain system level datausable by multiple tenants or other data. Such system level data mightinclude industry reports, news, postings, and the like that are sharableamong tenants.

In certain embodiments, user systems 812 (which may be client systems)communicate with application servers 900 to request and updatesystem-level and tenant-level data from system 816 that may requiresending one or more queries to tenant data storage 822 and/or systemdata storage 824. System 816 (e.g., an application server 900 in system816) automatically generates one or more structured query language (SQL)statements (e.g., one or more SQL queries) that are designed to accessthe desired information. System data storage 824 may generate queryplans to access the requested data from the database.

Each database can generally be viewed as a collection of objects, suchas a set of logical tables, containing data fitted into predefinedcategories. A “table” is one representation of a data object, and may beused herein to simplify the conceptual description of objects and customobjects according to the embodiments described herein. It should beunderstood that “table” and “object” may be used interchangeably herein.Each table generally contains one or more data categories logicallyarranged as columns or fields in a viewable schema. Each row or recordof a table contains an instance of data for each category defined by thefields. For example, a CRM database may include a table that describes acustomer with fields for basic contact information such as name,address, phone number, fax number, etc. Another table might describe apurchase order, including fields for information such as customer,product, sale price, date, etc. In some multi-tenant database systems,standard entity tables might be provided for use by all tenants. For CRMdatabase applications, such standard entities might include tables forAccount, Contact, Lead, and Opportunity data, each containingpre-defined fields. It should be understood that the word “entity” mayalso be used interchangeably herein with “object” and “table”.

In some multi-tenant database systems, tenants may be allowed to createand store custom objects, or they may be allowed to customize standardentities or objects, for example by creating custom fields for standardobjects, including custom index fields. In certain embodiments, forexample, all custom entity data rows are stored in a single multi-tenantphysical table, which may contain multiple logical tables perorganization. It is transparent to customers that their multiple“tables” are in fact stored in one large table or that their data may bestored in the same table as the data of other customers.

FIG. 10 illustrates a block diagram of sub-tenant process spaces 904-1 .. . 904-N that can be defined within one of the tenant process spaces904 of FIG. 9 to explain the concept of sub-multi-tenancy in accordancewith an embodiment. Further defining sub-tenant process spaces 904-1 . .. 904-N within one of the tenant process spaces 904 can be thought of assub multi-tenancy within a tenant of a multi-tenant system. Thedisclosed embodiments can address challenges posed by multi-tenantenvironment in which multiple sub-tenants (e.g., teams, business unitsor functional groups) use common space to store their test results andcode coverage metrics.

For example, in a cloud-based system that provides a multi-tenantapplication that is being tested, each manifest file can include astatically defined set of test class names each having a correspondingtest class identifier that identifies a test class to be executed, andeach regular expression is a pattern that is used to perform a patternmatch to identify a particular group of one or more test class namesthat correspond to one or more test classes associated with that regularexpression. The user specifies particular manifest file(s) and regularexpression(s) to define a targeted set/group of test class names thatare of interest to a particular sub-tenant (or group of sub-tenants) ofthe multi-tenant application during testing of the multi-tenantapplication. Stated differently, the filtered test class identifiersspecified via the unique test class identifier array correspond to asubset of all possible test classes that have been defined by theparticular sub-tenant (or group of sub-tenants) within a tenant ofmulti-tenant cloud application, and the subset of all possible testclasses are those that are of interest to that particular sub-tenant (orgroup of sub-tenants) and correspond to the particular subset of tests(of all possible tests that could be executed) that are to be executedduring testing of the multi-tenant application. In this way, the set oftest class names collectively represent a particular, targeted subset oftests that are of interest or pertain to that particular sub-tenant (orgroup of sub-tenants) within a tenant of multi-tenant cloud application,such as a specific business unit or functional group. By providing eachuser with the ability to group and filter test classes, a particularsub-tenant (within a tenant of multi-tenant cloud application) canselectively define which test classes are of interest, so that theparticular sub-tenant can select a particular subset of targeted teststhat are to be executed to generate only those target test results thatare of interest or pertinent to that particular sub-tenant.

Similarly, by providing each user with the ability to group and filtersource classes, a particular sub-tenant can selectively define whichsource classes are of interest, so that the particular sub-tenant cancompute code coverage metrics for a particular subset of targeted sourceclasses (of all possible source classes) to generate targeted codecoverage metrics that are of interest or pertinent to that particularsub-tenant. For instance, the targeted code coverage metrics can be aparticular subset of code coverage metrics that are of interest orpertinent to a specific group such as a specific sub-tenant or group ofsub-tenants, such as a specific business unit or functional group.

Any suitable programming language can be used to implement the routinesof particular embodiments including C, C++, Java, assembly language,etc. Different programming techniques can be employed such as proceduralor object oriented. The routines can execute on a single processingdevice or multiple processors. Although the steps, operations, orcomputations may be presented in a specific order, this order may bechanged in different particular embodiments. In some particularembodiments, multiple steps shown as sequential in this specificationcan be performed at the same time.

Particular embodiments may be implemented in a computer-readable storagemedium (also referred to as a machine-readable storage medium) for useby or in connection with the instruction execution system, apparatus,system, or device. Particular embodiments can be implemented in the formof control logic in software or hardware or a combination of both. Thecontrol logic, when executed by one or more processors, may be operableto perform that which is described in particular embodiments.

A “processor,” “processor system,” or “processing system” includes anysuitable hardware and/or software system, mechanism or component thatprocesses data, signals or other information. A processor can include asystem with a general-purpose central processing unit, multipleprocessing units, dedicated circuitry for achieving functionality, orother systems. Processing need not be limited to a geographic location,or have temporal limitations. For example, a processor can perform itsfunctions in “real time,” “offline,” in a “batch mode,” etc. Portions ofprocessing can be performed at different times and at differentlocations, by different (or the same) processing systems. A computer maybe any processor in communication with a memory. The memory may be anysuitable processor-readable storage medium, such as random-access memory(RAM), read-only memory (ROM), magnetic or optical disk, or othertangible media suitable for storing instructions for execution by theprocessor.

Particular embodiments may be implemented by using a programmed generalpurpose digital computer, by using application specific integratedcircuits, programmable logic devices, field programmable gate arrays,optical, chemical, biological, quantum or nanoengineered systems,components and mechanisms may be used. In general, the functions ofparticular embodiments can be achieved by any means as is known in theart. Distributed, networked systems, components, and/or circuits can beused. Communication, or transfer, of data may be wired, wireless, or byany other means.

It will also be appreciated that one or more of the elements depicted inthe drawings/figures can also be implemented in a more separated orintegrated manner, or even removed or rendered as inoperable in certaincases, as is useful in accordance with a particular application. It isalso within the spirit and scope to implement a program or code that canbe stored in a machine-readable medium to permit a computer to performany of the methods described above.

As used in the description herein and throughout the claims that follow,“a”, “an”, and “the” includes plural references unless the contextclearly dictates otherwise. Also, as used in the description herein andthroughout the claims that follow, the meaning of “in” includes “in” and“on” unless the context clearly dictates otherwise.

The foregoing description is merely illustrative in nature and is notintended to limit the embodiments of the subject matter or theapplication and uses of such embodiments. Furthermore, there is nointention to be bound by any expressed or implied theory presented inthe technical field, background, or the detailed description. As usedherein, the word “exemplary” means “serving as an example, instance, orillustration.” Any implementation described herein as exemplary is notnecessarily to be construed as preferred or advantageous over otherimplementations, and the exemplary embodiments described herein are notintended to limit the scope or applicability of the subject matter inany way.

For the sake of brevity, conventional techniques related to computerprogramming, computer networking, and other functional aspects of thesystems (and the individual operating components of the systems) may notbe described in detail herein. In addition, those skilled in the artwill appreciate that embodiments may be practiced in conjunction withany number of system and/or network architectures, data transmissionprotocols, and device configurations, and that the system describedherein is merely one suitable example. Furthermore, certain terminologymay be used herein for the purpose of reference only, and thus is notintended to be limiting. For example, the terms “first”, “second” andother such numerical terms do not imply a sequence or order unlessclearly indicated by the context.

Embodiments of the subject matter may be described herein in terms offunctional and/or logical block components and with reference tosymbolic representations of operations, processing tasks, and functionsthat may be performed by various computing components or devices. Suchoperations, tasks, and functions are sometimes referred to as beingcomputer-executed, computerized, software-implemented, orcomputer-implemented. In this regard, it should be appreciated that thevarious block components shown in the figures may be realized by anynumber of hardware, software, and/or firmware components configured toperform the specified functions.

For example, an embodiment of a system or a component may employ variousintegrated circuit components, e.g., memory elements, digital signalprocessing elements, logic elements, look-up tables, or the like, whichmay carry out a variety of functions under the control of one or moremicroprocessors or other control devices. In this regard, the subjectmatter described herein can be implemented in the context of anycomputer-implemented system and/or in connection with two or moreseparate and distinct computer-implemented systems that cooperate andcommunicate with one another.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or embodiments described herein are not intended tolimit the scope, applicability, or configuration of the claimed subjectmatter in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the described embodiment or embodiments. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the scope defined by theclaims, which includes known equivalents and foreseeable equivalents atthe time of filing this patent application. Accordingly, details of theexemplary embodiments or other limitations described above should not beread into the claims absent a clear intention to the contrary.

What is claimed:
 1. A system, comprising: a user system, comprising: aninput system configured to receive input parameters specified by a userof the user system, wherein the input parameters comprise: one or moreregular expressions specified by the user of the user system, whereineach regular expression is a pattern that is used to perform a patternmatch to identify a particular group of one or more first test classidentifiers associated with that regular expression that each correspondto a test class name of a test class; a processing system; and a memoryconfigured to store: a test class filter module executable by theprocessing system, wherein the test class filter module, upon beingexecuted by the processing system, is configured to: group and filtertest class identifiers, based on one or more of the input parameters, togenerate a unique test class identifier array of filtered test classidentifiers that correspond to a particular subset of tests that are tobe executed during testing of an application, wherein the test classfilter module comprises: a first test class fetcher comprising: aregular expression processor configured to receive the regularexpressions input by the user of the user system; and a query builderconfigured to: generate and send a query associated with each regularexpression to an interface, wherein the interface is configured toreceive each query, contact a data store and find matching test classnames for each regular expression, and return a first test classidentifier for each of the matching test class names to the querybuilder, and wherein the query builder is further configured to receivethe first test class identifiers for each of the matching test classnames, and group the first test class identifiers corresponding to eachof the regular expressions into a first test class identifier array,wherein the test class filter module further comprises: a unique classID filter configured to: receive the first test class identifier arrayand a second test class identifier array; and filter the one or morefirst test class identifiers from the first test class identifier arrayand second test class identifiers from the second test class identifierarray to generate the unique test class identifier array of filteredtest class identifiers by: combining the first test class identifiersfrom the first test class identifier array and the second test classidentifiers from the second test class identifier array into a combinedset of test class identifiers; determining duplicate test classidentifiers that are in the combined set; determining invalid test classidentifiers that are in the combined set; and eliminating the duplicatetest class identifiers and the invalid test class identifiers from thecombined set to generate the unique test class identifiers array offiltered test class identifiers that correspond to the particular subsetof tests that are to be executed; and a cloud-based computing platform,communicatively coupled to the user system, and comprising: a testexecution engine configured to execute the particular subset of testscorresponding to the filtered test class identifiers specified in theunique test class identifier array.
 2. The system according to claim 1,wherein the cloud-based computing platform further comprises: theinterface that interfaces with the test class filter module, wherein theinput parameters further comprise: a manifest file specified by the userof the user system, and wherein the test class filter module furthercomprises: a second test class fetcher comprising: a manifest fileprocessor configured to: retrieve the manifest file specified by theuser of the user system, wherein the manifest file comprises a pluralityof test class names that can be used to fetch a particular group of oneor more second test class identifiers from the data store of thecloud-based computing platform; and a manifest file reader and querybuilder configured to: read the test class names from the manifest file,send the test class names from the manifest file to the interface,receive the one or more second test class identifiers corresponding toeach of the test class names from the manifest file from the interface,and group the second test class identifiers into the second test classidentifier array.
 3. The system according to claim 2, wherein themanifest file has a name, and wherein the plurality of test class namesin the manifest file comprises: a statically defined set of test classnames each having a corresponding second test class identifier thatidentifies a test class to be executed.
 4. The system according to claim3, wherein the application is a multi-tenant application, and whereinthe user selects the manifest file to specify the statically defined setof test class names that are of interest to a particular sub-tenant ofthe multi-tenant application during testing of the multi-tenantapplication.
 5. The system according to claim 1, wherein the applicationis a multi-tenant application, and wherein the user selects the one ormore regular expressions to specify a particular group of one or moretest class names that are of interest to a particular sub-tenant of themulti-tenant application during testing of the multi-tenant application.6. The system according to claim 1, wherein the unique test classidentifier array of filtered test class identifiers correspond to aparticular subset of all possible test classes that have been defined bythe user via the input parameters.
 7. The system according to claim 6,wherein the application is a multi-tenant application, wherein the useris a sub-tenant, and wherein the particular subset of all possible testclasses are those that are of interest to that particular sub-tenant andcorrespond to the particular subset of tests that are to be executedduring testing of the multi-tenant application.
 8. The system accordingto claim 1, wherein the memory is further configured to store a testmanagement module comprising: a test enqueuer module executable by theprocessing system, wherein the test enqueuer module, upon being executedby the processing system, is configured to: receive the unique testclass identifier array of filtered test class identifiers from the testclass filter module; and communicate the filtered test class identifiersas a batch to the test execution engine, wherein the test executionengine is configured to execute the particular subset of testscorresponding to each of the filtered test class identifiers specifiedin the unique test class identifier array; and a test resultpoller/fetcher executable by the processing system, wherein the testresult poller/fetcher, upon being executed by the processing system, isconfigured to: receive, from the test execution engine, a parent jobidentifier used to track test execution status of the particular subsetof tests corresponding to each of the filtered test class identifiersthat are to be executed at the test execution engine; use the parent jobidentifier to poll the test execution engine to monitor the progress ofthe particular subset of tests; and when the test result poller/fetcherdetermines that all tests in the particular subset of tests have beenexecuted by the test execution engine: communicate the parent jobidentifier to the interface to request test results and triggerretrieval of the test results from the data store.
 9. The systemaccording to claim 8, wherein the input parameters comprise: a testexecution time limit specified by the user of the user system, whereinthe test enqueuer module is configured to receive the test executiontime limit, and wherein the test management module further comprises: atime tracker configured to: monitor an amount of time each test takes toexecute, and issue an instruction to automatically stop any test if theamount of time that the test executes exceeds the test execution timelimit for completion of that test.
 10. The system according to claim 8,wherein the memory is further configured to store: a test reportgenerator executable by the processing system, wherein the test reportgenerator, upon being executed by the processing system, is configuredto: receive the test results from the interface; and generate a testreport comprising: the test results for the particular subset of testscorresponding to the unique test class identifier array of filtered testclass identifiers.
 11. A method for executing tests in a cloud-basedcomputing system comprising a user system and a cloud-based computingplatform communicatively coupled to the user system via an interface ofthe cloud-based computing platform, the method comprising: receivinginput parameters specified by a user form an input system of the usersystem, wherein the input parameters comprise: a manifest file specifiedby the user of the user system, and one or more regular expressionsspecified by the user of the user system, wherein each regularexpression is a pattern that is used to perform a pattern match toidentify a particular group of one or more first test class identifiersassociated with that regular expression that each correspond to a testclass name of a test class; at a processing system of the user system:grouping and filter test class identifiers, based on one or more of theinput parameters, to generate a unique test class identifiers array offiltered test class identifiers that correspond to a particular subsetof tests that are to be executed during testing of an application,wherein the grouping and filtering test class identifiers, furthercomprises: retrieving, via a manifest file processor, the manifest file,wherein the manifest file comprises a plurality of test class names thatcan be used to fetch a particular group of one or more second test classidentifiers from a data store of the cloud-based computing platform;retrieving, via a regular expression processor, the one or more regularexpressions; at a query builder: generating a query associated with eachof the one or more regular expressions; at the interface: receiving eachquery generated by the query builder; contacting the data store andfinding matching test class names for each regular expression; andreturning a first test class identifier for each of the matching testclass names to the query builder; at the query builder: receiving theone or more first test class identifiers of each of the matching testclass names, and grouping the one or more first test class identifierscorresponding to each of the regular expressions into a first test classidentifier array; at a manifest file reader and query builder: readingthe test class names from the manifest file; sending the test classnames from the manifest file to the interface; receiving, from theinterface, the particular group of the one or more second test classidentifiers corresponding to each of the test class names from themanifest file; and grouping the particular group of the one or moresecond test class identifiers into a second test class identifier array;filtering, at a unique class ID filter, the one or more first test classidentifiers from the first test class array and the one or more secondtest class identifiers from the second test class identifier array togenerate the unique test class identifier array of filtered test classidentifiers, wherein the unique test class identifier array of thefiltered test class identifiers corresponds to a subset of test classesthat have been defined based on the input parameters: and at a testexecution engine of the cloud-based computing platform: executing theparticular subset of tests corresponding to the filtered test classidentifiers of the unique test class identifier array.
 12. The methodaccording to claim 11, wherein the application is a multi-tenantapplication, and wherein the plurality of test class names in themanifest file comprise: a statically defined set of test class nameseach having a corresponding test class identifier that identifies a testclass to be executed, wherein the user selects the manifest file tospecify the statically defined set of test class names that are ofinterest to a particular sub-tenant of the multi-tenant applicationduring testing of the multi-tenant application.
 13. The method accordingto claim 11, wherein the user selects the one or more regularexpressions to specify a particular group of one or more test classnames that are of interest to a particular sub-tenant of themulti-tenant application during testing of the multi-tenant application.14. The method according to claim 11, wherein filtering comprises:combining the one or more first test class identifiers from the firsttest class identifier array and the one or more second test classidentifiers from the second test class identifier array into a combinedset of test class identifiers; determining duplicate test classidentifiers that are in the combined set; determining invalid test classidentifiers that are in the combined set; and eliminating the duplicatetest class identifiers and the invalid test class identifiers from thecombined set to generate the unique test class identifier array offiltered test class identifiers that correspond to the particular subsetof tests that are to be executed.
 15. The method according to claim 11,wherein the method further comprises: at a test management module of theuser system: receiving the unique test class identifier array offiltered test class identifiers; communicating the filtered test classidentifiers as a batch to the test execution engine; receiving, from thetest execution engine, a parent job identifier used to track status oftest execution of the particular subset of tests corresponding to eachof the filtered test class identifiers that are to be executed at thetest execution engine; and using the parent job identifier to poll thetest execution engine to monitor the progress of the particular subsetof tests; and when all tests in the particular subset of tests have beenexecuted by the test execution engine: communicating the parent jobidentifier to the interface to request test results and triggerretrieval of the test results from the data store.
 16. The methodaccording to claim 15, wherein the method further comprises: at a testreport generator: receiving the test results from the interface; andgenerating a test report comprising: the test results for the particularsubset of tests corresponding to the unique test class identifier arrayof filtered test class identifiers.
 17. A computer program product,comprising: a non-transitory computer readable medium having a computerreadable program code embodied therein that is readable by a processingcircuit and storing instructions for execution by the processing circuitto perform a method, the method comprising: receiving input parametersspecified by a user from an input system of a user system, wherein theinput parameters comprise: one or more manifest files specified by theuser of the user system each comprising a plurality of test class namesthat can be used to fetch a particular group of one or more first testclass identifiers from a data store of a cloud-based computing platform,and one or more regular expressions specified by the user of the usersystem, wherein each regular expression is a pattern that is used toperform a pattern match to identify a particular group of one or moresecond test class identifiers associated with that regular expressionthat each correspond to a test class name of a test class; grouping andfiltering test class identifiers, based on one or more of the inputparameters, to generate a unique test class identifier array of filteredtest class identifiers that correspond to a particular subset of teststhat are to be executed by a test execution engine during testing of anapplication, wherein the grouping and filtering test class identifiers,further comprises: retrieving, via a manifest file processor, themanifest file, wherein the manifest file comprises a plurality of testclass names that can be used to fetch the particular group of the one ormore first test class identifiers from the data store of the cloud-basedcomputing platform; at a manifest file reader and query builder readingthe test class names from the manifest file; sending the test classnames from the manifest file to the interface; receiving, from theinterface, the particular group of the one or more first test classidentifiers corresponding to each of the test class names from themanifest file; and grouping the particular group of the one or morefirst test class identifiers into a first test class identifier array;retrieving, via a regular expression processor, the one or more regularexpressions; at a query builder, generating a query associated with eachof the one or more regular expressions; at the interface receiving eachquery generated by the query builder; contacting the data store andfinding matching test class names for each regular expression; andreturning a second test class identifier for each of the matching testclass names to the query builder; at the query builder: receiving theone or more second test class identifiers for each of the matching testclass names, and grouping the one or more second test class identifierscorresponding to each of the regular expression into a second test classidentifier array; filtering, at a unique class ID filter, the one ormore first test class identifiers from the first test class identifierarray and the one or more second test class identifiers from the secondtest class identifier array to generate the unique test class identifierarray of filtered test class identifiers, wherein the unique test classidentifier array of the filtered test class identifiers corresponds to asubset of test classes that have been defined based on the inputparameters; and executing, at a test execution engine of the cloud-basedcomputing platform, the particular subset of tests corresponding to thefiltered test class identifiers specified in the unique test classidentifier array.